Emotion Detection of People Wearing Face Mask

Though many facial emotion recognition models exist, after the Covid-19 pandemic, majority of such algorithms are rendered obsolete as everybody is compelled to wear a facemask to protect themselves against the deadly virus. Face masks can hinder emotion recognition systems, as crucial facial features are not visible in the image. This is because facemasks cover essential parts of the face such as the mouth, nose, and cheeks which play an important role in differentiating between various emotions. This study intends to recognize the emotional states of anger-disgust, neutral, surprise-fear, joy, sadness, of the person in the image with a face mask. In the proposed method, a CNN model is trained using images of people wearing masks. To achieve higher accuracy, the classes in the dataset are combined. Different combinations of clubbing are performed, and results are recorded. Images are taken from FER2013 dataset which consists of a huge number of manually annotated facial images of people. © 2023 IEEE.

Engagement in Video Graphic Online Learning Using the Emotional Dimensions in the Learning Context

The COVID-19 pandemic has become the prime reason for organizations across the world to shift their entire workforce onto virtual platforms. One of the major drawbacks of these virtual platforms is that it lacks a real-time metric which could be used to detect whether a person is attentive during the lectures and meetings or not. This was most evident in the case of educational institutions, where students would often fail to pay attention to the content that was being taught by teachers and professors at home. With this research work, our aim is to create a solution for this problem with the help of AI-FER (Artificial Intelligence Facial Emotion Recognition). For this, we have proposed our own Convolutional Neural Network model achieving an overall accuracy of 59.03%. We have also used several pre-trained models available in Google's Tensorflow library like DenseNET and VGG. © 2023 IEEE.

Poetry in Pandemic: A Multimodal Neuroaesthetic Study on the Emotional Reaction to the Divina Commedia Poem

Poetry elicits emotions, and emotion is a fundamental component of human ontogeny. Although neuroaesthetics is a rapidly developing field of research, few studies focus on poetry, and none address its different modalities of fruition (MOF) of universal cultural heritage works, such as the Divina Commedia (DC) poem. Moreover, alexithymia (AX) resulted in being a psychological risk factor during the COVID-19 pandemic. The present study aims to investigate the emotional response to poetry excerpts from different cantica (Inferno, Purgatorio, Paradiso) of DC with the dual objective of assessing the impact of both the structure of the poem and MOF and that of the characteristics of the acting voice in experts and non-experts, also considering AX. Online emotion facial coding biosignal (BS) techniques, self-reported and psychometric measures were applied to 131 literary (LS) and scientific (SS) university students. BS results show that LS globally manifest more JOY than SS in both reading and listening MOF and more FEAR towards Inferno. Furthermore, LS and SS present different results regarding NEUTRAL emotion about acting voice. AX influences listening in NEUTRAL and SURPRISE expressions. DC's structure affects DISGUST and SADNESS during listening, regardless of participant characteristics. PLEASANTNESS varies according to DC's structure and the acting voice, as well as AROUSAL, which is also correlated with AX. Results are discussed in light of recent findings in affective neuroscience and neuroaesthetics, suggesting the critical role of poetry and listening in supporting human emotional processing. © 2023 by the authors.

Student Attention Base Facial Emotion State Recognition Using Convolutional Neural Network

E-learning system advancements give students new opportunities to better their academic performance and access e-learning education. Because it provides benefits over traditional learning, e-learning is becoming more popular. The coronavirus disease pandemic situation has caused educational institution cancelations all across the world. Around all over the world, more than a billion students are not attending educational institutions. As a result, learning criteria have taken on significant growth in e-learning, such as online and digital platform-based instruction. This study focuses on this issue and provides learners with a facial emotion recognition model. The CNN model is trained to assess images and detect facial expressions. This research is working on an approach that can see real-time facial emotions by demonstrating students' expressions. The phases of our technique are face detection using Haar cascades and emotion identification using CNN with classification on the FER 2013 datasets with seven different emotions. This research is showing real-time facial expression recognition and help teachers adapt their presentations to their student's emotional state. As a result, this research detects that emotions' mood achieves 62% accuracy, higher than the state-of-the-art accuracy while requiring less processing. © 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

Investigation on MLP, CNNs and Vision Transformer models performance for Extracting a Human Emotions via Facial Expressions

One of the difficult challenges in AI development is to make machine understand the human feeling through expression because human can express feeling in various ways, for example, through voices, facial actions or behaviors. Facial Emotion Recognition (FER) has been used in interrogating suspects and being a tool to help detect emotions in people with nerve damage or even in the COVID-19 pandemic when patients hide their timelines. It can be applied to detect lies through micro expression. In this work will mainly focus on FER. The results of Deep Neural Network (DNN), Convolutional Neural Network (CNN), and Vision Transformer were compared. Human emotion expressions were classified by using facial expression datasets from AffectNet, Tsinghua, Extended Cohn Kanade (CK+), Karolinska Directed Emotional Faces (KDEF) and Real-world Affective Faces (RAF). Finally, all models were evaluated on the testing dataset to confirm their performance. The result shows that Vision Transformer model outperforms other models. © 2023 IEEE.

A user-oriented multi-input Sentiment Analysis and Feedback System for mental state awareness necessitated due to COVID-19

Development in technology has led to a spike in sharing of opinions about different subjects on social media, for instance, movie or product reviews. Unprecedented COVID-19 led to forced isolation and affected mental health negatively. This paper introduces a system to detect users' emotions and mental states based on provided input. Among the different data sources available on social media, real-time Twitter data is used in this analysis. Sentiment analysis can be used as a tool at various levels, right from individual to organizational development. Deep learning algorithms like LSTM and CNN lay the foundation of this system. Python libraries and Google APIs are used to add functionalities. Earlier studies only focused on detecting emotions, whereas the proposed system provides the user with a graphical analysis of detected emotions and apt suggestions like motivational quotes or videos. The system accepts multilingual text input, speech, or video input. The scope of this system is not restricted to COVID-19 related texts. This research will assist individuals and businesses and aid future development. © 2022 IEEE.

A Comparative Survey of Multimodal Multilabel Sentiment Analysis and Its Applications Initiated Due to the Impact of COVID-19

This study presents a detailed survey of different works related to sentiment analysis. The COVID-19 pandemic and its impact on people's mental health act as the driving force behind this survey. The survey can help study sentiment analysis and approaches taken in many studies to detect human emotions via advanced technology. It can also help in improving present systems by finding loopholes and increasing their accuracy. Various lexicon and ML-based systems and models like Word2Vec and LSTM were studied in the surveyed papers. Some of the current and future directions highlighted were Twitter sentiment analysis, review-based market analysis, determining changing behavior and emotions in a given time period, and detecting the mental health of employees, and students. This survey provides details related to trends and topics in sentiment analysis and an in-depth understanding of various technologies used in different studies. It also gives an insight into the wide variety of applications related to sentiment analysis. © 2022 IEEE.

A Deep Learning Approach to Facial Expression Recognition in the Presence of Masked Occlusion

Detecting facial expressions is a vital aspect of interpersonal communication. Automatic facial emotion recognition (FER) systems for detecting and analyzing human behavior have been a subject of study for the past decade, and have played key roles in healthcare, crime detection, and other use cases. With the worldwide spread of the COVID-19 pandemic, wearing face masks while interacting in public spaces has become recommended behavior to protect against infection. Therefore, improving existing FER systems to tackle mask occlusion is an extremely important task. In this paper, we analyze how well existing CNN models for FER fare with masked occlusion and present deep CNN architectures to solve this task. We also test some methods to reduce model overfitting, such as data augmentation and dataset balancing. The main metric used to compare the models is accuracy, and the dataset used here is FER2013. Images from FER2013 were covered by masks using a certain transformation, resulting in a new dataset, MFER2013. From our evaluation and experimentation, we found that existing models need to be modified before they can achieve good accuracy on masked datasets. By improving the architecture of the base CNN, we were able to achieve a significantly improved accuracy. © 2022 IEEE.

Adaptive Learning Systems based on ILOs of Courses

Nowadays, the use of e-learning techniques and methods is a very important challenge due to the importance of digital transformation to all countries. Firstly, the spread of the COVID-19 virus all over the world. Secondly, all students need to study their courses remotely from home to reduce the communication with others to save their life. All teachers need to engage their students effectively to study an online course, get more knowledge and high results at the end of these courses. Data mining is the best tool used to find a hidden pattern. We used an educational data mining tool to help teachers find the pros and cons of using an e-learning course with their students. We need to classify students on these online courses according to their ability to understand materials and quizzes, or assessment methods of the course, by making adaptive e-learning courses. In this paper, we will show the importance of using adaptive e-learning courses and the challenges faced by authors to build these systems, and we will list the different methods used with adaptive learning like gamification, brain-hex models, facial emotions, and we will also list a survey about other authors' techniques and methods used to find the student's learner style. We build a new proposed model of ILOs(Intended Learning Outcomes) adaptive learning with the emotion-based system to let the system find the student's learning style and build the material according to their skills and knowledge outcomes from the course and engage the use of facial emotion while taking the quiz to predict the student's results and the topics he/she needs to study more via our system to achieve high grades and knowledge. Our system finds that the visual students have the highest grades with 75%, followed by kinesthetic with 70% and the lowest grades in auditory with 50%. © 2023, World Scientific and Engineering Academy and Society. All rights reserved.

Adaptive Learning Systems based on ILOs of Courses

Nowadays, the use of e-learning techniques and methods is a very important challenge due to the importance of digital transformation to all countries. Firstly, the spread of the COVID-19 virus all over the world. Secondly, all students need to study their courses remotely from home to reduce the communication with others to save their life. All teachers need to engage their students effectively to study an online course, get more knowledge and high results at the end of these courses. Data mining is the best tool used to find a hidden pattern. We used an educational data mining tool to help teachers find the pros and cons of using an e-learning course with their students. We need to classify students on these online courses according to their ability to understand materials and quizzes, or assessment methods of the course, by making adaptive e-learning courses. In this paper, we will show the importance of using adaptive e-learning courses and the challenges faced by authors to build these systems, and we will list the different methods used with adaptive learning like gamification, brain-hex models, facial emotions, and we will also list a survey about other authors' techniques and methods used to find the student's learner style. We build a new proposed model of ILOs(Intended Learning Outcomes) adaptive learning with the emotion-based system to let the system find the student's learning style and build the material according to their skills and knowledge outcomes from the course and engage the use of facial emotion while taking the quiz to predict the student's results and the topics he/she needs to study more via our system to achieve high grades and knowledge. Our system finds that the visual students have the highest grades with 75%, followed by kinesthetic with 70% and the lowest grades in auditory with 50%. © 2023, World Scientific and Engineering Academy and Society. All rights reserved.

Support of Virtual Human Interactions Based on Facial Emotion Recognition Software

Emotion recognition based on facial expressions is an increasingly important area in Human-Computer Interaction Research. Despite the many challenges of computer-based facial emotion recognition like, e.g., the huge variability of human facial features, cultural differences, and the differentiation between primary and secondary emotions, there are more and more systems and approaches focusing on facial emotion recognition. These technologies already offer many possibilities to automatically recognize human emotions. As part of a research project described in this paper, these technologies are used to investigate whether and how they can support virtual human interactions. More and more meetings are taking place virtually due to the Covid-19 pandemic and the advancing digitalization. Therefore, the face of the attendees is often the only visible part that indicates emotional states. This paper focuses on outlining why emotions and their recognition are important and in which areas the use of automated emotion detection tools seems to be promising. We do so by showing potential use cases for visual emotion recognition in the professional environment. In a nutshell, the research project aims to investigate whether facial emotion recognition software can help to improve self-reflection on emotions and the quality and experience of virtual human interactions. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Using COVID-19 Masks Dataset to Implement Deep Convolutional Neural Networks For Facial Emotion Recognition

In light of the novel pandemic called COVID-19, the world has been instructed to wear protective facial masks to limit its spread. Doing so has reduced the effectiveness of traditional facial recognition technologies, especially in processing human facial emotions. This has rendered the usage of such technology obsolete in managing facial databases, relying on it for security purposes, and so on. It is then necessary to enhance the current generation of facial recognition to adapt to the protective masks. Speaking of the current facial recognition generation, most of its complex iterations heavily rely on deep learning, which is flawed since the existing facial databases are insufficient, making it even more inadequate to bypass facial masks. This is why the present research paper suggests implementing the Deep Convolutional Neural Networks (DCNN) algorithm using the Japanese Female Facial Expression (JAFFE) to simulate a masked face emotion recognition. This facial database is available free for academic research, was utilized to label the available images displaying various facial emotions under the umbrella of one of the seven basic human facial emotions, allowing for a more advanced facial technology. Consistent with the latest research findings, the proposed facial emotion recognition attains up to an accuracy of 71.35% due to its meticulous masked facial database. © 2021 IEEE.

Sensor-Based Emotion Tracking System for Computer Games

The game development industry is among the leading industries globally, and in 2020, gaming emerged as a popular entertainment activity upon the COVID-19 outbreak. Thus, competition among gaming companies is high. Hence, they try to adopt new technologies often. Gaming brings multiple feelings for the gamer. At times, the conditions may get even worse from the game's end where the gamer may end up venting out his rage and annoyance. Hence, there is a massive possibility for the gamer to switch to another game which may result in the company to lose its customers. In that scenario, this system can monitor the emotional states of the gamer while playing and manipulate the gaming environment, sound environment, enemy behavior, and gamer mechanism according to the emotional state of the gamer. The sensor-based emotion tracking system identifies the gamer's emotional state using facial emotions, detected through a webcam and heart rate, detected through sensors. The development was carried out through the machine learning models, open cv, Arduino techniques, and reactive programming. The emotional state and facial emotions that will be tracked will count to an accuracy of above 95%. Through that, the target will be to make the gamer satisfied by building appreciation for the services given and by improving the gamer's gaming experience and retain the gamer with the game provider. © 2021 IEEE.